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Abstract 

Background: Computer reminders are increasingly being applied in efforts to improve quality and patient safety. 
However, research is still needed to establish the effectiveness of different kinds of reminders in various settings. 
This study aimed to evaluate the effectiveness of computer reminders for improving adherence to a quality 
assessment scheme for point-of-care testing in general practice. 

Method: The study was conducted as a randomized controlled crossover trial among general practices in the 
Capital Region of Denmark. The intervention consisted of sending computer reminders (ComRem) to practices not 
adhering to the guideline recommendations of split testing for hemoglobin and glucose. Practices were randomly 
allocated into two groups. During the first follow-up period, one of the groups received the ComRem intervention 
together with the general implementation activities (GIA), while the other group only received the GIA. For the 
second follow-up period, the intervention was switched between the two groups. Outcomes were measured as 
split test procedure adherence. 

Results: A total of 142 practices were randomly allocated to the early intervention group and 144 practices to the 
late intervention group (the control group in the first follow-up period). In the first intervention period, the mean 
number of split tests performed in the group receiving ComRem group increased from 1.22 to 3.76 (out of eight 
possible tests) while the mean number of split tests increased from 1.1 1 to 2.35 in the group targeted by GIA only 
(p = 0.0059). After the crossover, a similar effect of reminders was observed. Furthermore, the developments in 
outcome measures over time showed a strong effect of computer reminders beyond the intervention periods. 

Conclusion: There was a significant effect of computer reminders on adherence to the quality assessment scheme 
for point-of-care testing. Thus, computer reminders seem to be useful for supporting the implementation of 
relatively simple procedures for quality and safety. 

Trial registration: ClinicalTrials.gov: NCT01 152177 
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Background 

Point-of-care testing (POCT) is increasingly being emplo- 
yed in general practice to improve diagnostic capabilities 
and the delivery of timely patient care [1-4]. Since POCT 
results influence the daily decisions of GPs, it is clearly im- 
portant that the results are accurate. Thus the quality of 
care will suffer from misleading POCT results; in worst 
case, inaccurate results may cause the general practitioner 
(GP) to overlook life-threatening conditions such as 
hypoglycemic incidences. In order to ensure the technical 
and professional quality of POCT in general practice, 
quality assessment schemes have been set up in several 
countries [5,6], and split testing has been found to be a 
cost-effective way of performing external quality assess- 
ment [7]. In the Capital Region of Denmark, external qua- 
lity assessment is enforced through a split test procedure 
and an annual visit by a facilitator from the Copenhagen 
General Practitioners Laboratory ('the Laboratory'). How- 
ever, adherence to the monthly split test procedure has 
been low among the GPs in the Capital Region. Thus, pre- 
vious to this study, one-half of the clinics did not perform 
the required split tests in the baseline period and no 
clinics performed more than three quarters of the possible 
number of tests (Table 1). Therefore, the Laboratory and 
the administration of the Capital Region planned to im- 
prove adherence by using computer reminders (ComRem) 



embedded in the GPs' electronic medical records (in 
addition to the general activities to promote quality assess- 
ment). At the same time, the parties wanted to carry out a 
thorough evaluation of this new implementation method. 

Within the last ten years, several systematic reviews 
have assessed the effects of computer reminders in regards 
to changing provider behavior [8-11]. The most recent 
Cochrane review finds the effects of computer reminders 
to be small to moderate, but due to the variation between 
the different types of reminder interventions and their re- 
sults, the review concludes that more 'research is needed 
to identify what types of reminders work and when' [11], 
p. 2. On this background, the purpose of this randomized 
controlled trial (RCT) was to assess the effectiveness of 
ComRem for improving adherence to the quality assess- 
ment scheme for POCTs. 

Methods 

Study participants 

The study took place in the Capital Region of Denmark in 
2010 and 2011. A detailed description of the study protocol 
has been presented elsewhere [12]. A total of 567 practices 
with a total of 739 GPs were eligible in the study area with 
a total population of 1.1 million. Only general practices 
conducting at least five POCTs for either hemoglobin or 
glucose — and practices that did not conduct POCT for 



Table 1 Baseline data (January to April 2010): GP and practice characteristics and distribution of point-of-care tests in 
the two RCT groups 





Total 


Early ComRem 


Late ComRem 


GP characteristics' 3 


(n = 341) 


(n = 1 70) 


(n = 171) 


Gender, n (%) 








Male 


171 (50.2) 


79 (46.5) 


92 (53.8) 


Female 


1 70 (49.8) 


91 (53.5) 


79 (46.2) 


Age, mean (SD) 


56.3 (8.2) 


55.8 (8.2) 


56.8 (8.2) 


Years as GP, mean (SD) 


14.8 (9.6) 


14.8 (94) 


14.9 (9.8) 


Practice characteristics 


(n = 286) 


(n = 142) 


(n = 144) 


Practice organization, n (%) 








Single handed 


239 (83.6) 


1 1 9 (83.8) 


120 (83.3) 


Group 


47 (16.4) 


23 (16.2) 


24 (16.7) 


Number of patients per doctor, mean (SD) 


1,599 (404) 


1,584 (395) 


1,614 (413) 


Performed point-of-care tests 








No. of Hemoglobin tests per practice, mean (SD) 


39.8 (54.7) 


41.5 (59.6) 


38.0 (49.4) 


No. of Glucose tests per practice, mean (SD) 


47.2 (52.4) 


52.1 (57.5) 


42.3 (46.5) 


Primary outcome 








No. of split tests (out of 8) per practice, mean (SD) 


1.11 (1.37) 


1.19 (144) 


1.03 (1.29) 


Secondary outcomes 








No. of practices with >0% of recommended split tests performed, n (%) 


146 (51.1) 


73 (51.4) 


73 (50.7) 


No. of practices with >75% of recommended split tests performed, n (%) 


0 (0.0) 


0 (0.0) 


0 (0.0) 



a P-value from a chi-squared test (categorical variables) or t-test (continuous variables), 
kfhe tests on GP characteristics account for the clustering of GPs in practices. 
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International Normalised Ratio (INR) — during the baseline 
period were to be included in the study. The reason for ex- 
cluding practices that conducted INR was that these prac- 
tices were to be included in another RCT on the effect of 
computer reminders versus postal reminders [12]. The 
practices were identified via the database of the Laboratory 
and the GP-database of the Capital Region. Because the 
intervention was part of the formal implementation activ- 
ities sanctioned by the Capital Region and the Laboratory, 
the sample in this study included all relevant practices irre- 
spective of their wish to participate. 

The split test procedure 

The quality assessment guidelines recommend a split test 
procedure in each practice each month for each POCT 
type. In the split test procedure, the POCT result is com- 
pared to the result of a blood sample (from the same pa- 
tient) analyzed at the Laboratory. The quotient of these two 
results should ideally be 1.00, but if the quotient lays within 
the interval 0.84 to 1.16 for hemoglobin and 0.82 to 1.18 
for glucose it is acceptable [13]. The result of the analysis is 
sent to the general practice for self-evaluation. If a practice 
does not respond to a case of unacceptable deviation by 
sending in a new split test, the Laboratory asks the practice 
to do so. If the result still deviates too much, the Labora- 
tory contacts the practice to help them locate the problem. 

General implementation activities (GIA) 

In 2010, the Laboratory generally stepped up its imple- 
mentation activities concerning the external quality assess- 
ment of POCT in general practice. First, the importance 
of performing split tests was emphasized at the annual fa- 
cilitator visit to each practice as well as in three laboratory 
newsletters. Second, a few technical changes were intro- 
duced that made it easier for the practices to order split 
tests (particularly, in April 2010, the ordering procedure 
was fully digitalized). During the intervention periods, the 
facilitators were not informed about the allocation status 
of the clinics prior to the visits. 

The intervention: computer reminders (ComRem) 

In the Danish healthcare sector, a common standard for 
secure electronic communication, MedCom, between 
healthcare providers has been in use for a decade. Within 
MedCom, each general practice has been allocated to a 
unique location number and all electronic communication 
is fed into the GPs electronic patient journal system. 
When the electronic journal system is accessed by the GP 
(or appointed staff), the system shows all communication 
that must be opened and approved. Hence, the computer 
reminders in this study are neither postal reminders nor 
traditional e-mails. 

The intervention consisted of sending computer re- 
minders (ComRem) to practices not adhering to the 



guideline recommendations of split testing for hemoglobin 
and glucose within the previous calendar month. Figure 1 
shows the content of the reminders. 

Randomization and design 

Practices were randomly allocated into two groups strati- 
fied by practice characteristics (group versus single 
handed) and geographic location. The randomization was 
done by means of computer-generated random numbers 
using SAS version 9.2. The randomization was performed 
by the data manager of the Research Unit of General Prac- 
tice without knowledge of the individual practice identifi- 
cation. The allocation procedure was concealed from the 
project group and was conducted by an independent 
organization, the Danish College of General Practitioners. 
At enrolment, the project group delivered practice codes 
and information regarding stratification to the college 
where a designated staff conducted the allocation and 
returned the information to project group. 

During the first follow-up period, one of the allocated 
groups (Early ComRem) received the ComRem interven- 
tion together with the GIA, while the other group (Late 
ComRem) received the GIA only. For the second follow- 
up period, the ComRem intervention was switched be- 
tween the two groups so that all practices would receive 
ComRem in the overall study period. This crossover was 
chosen in order to accommodate our scientific interest in 
having a control group that did not receive the interven- 
tion with the wishes of the Laboratory and the Capital Re- 
gion that all practices received the reminder intervention 
at some point during the study period in order to increase 
overall adherence. At the same time, the crossover design 
provided us with the opportunity to measure a legacy ef- 
fect of the intervention. 

Sample size 

To determine the power of the study, we used an estimate 
of a mean number of one hemoglobin and/or glucose split 
test based on laboratory data from 2007. Given a standard 
deviation of 1.25, a power of 90% and an effect of 0.5, we 
estimated that 266 practices were to be included in the 
study. We expected dropout rates to be negligible. Possible 
reasons for drop out were retirement or if a practice 
stopped using its own POCT equipment. 

Outcomes 

Outcomes were measured as split test procedure adher- 
ence, i.e., by the number of split tests received by the La- 
boratory. Outcomes were calculated and compared for 
three intervals of four months each: 

1. The comparison of outcomes after the first 
intervention period estimates the relative 
effectiveness of ComRem versus GIA only. 
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Copenhagen, DD.MM.YYYY 

Dear Colleague, 

Within the last calendar month we have not detected an external quality assessment analysis for 
haemoglobin or glucose from your practice point-of-care equipment. 

Execution of external quality assessment analysis is important for improving the quality of point-of-care 
testing and is a prerequisite for receiving fees for this service. 

Apply number XXXXXX-XXXX for ordering external quality assessment analysis in the WebQuality system. For 
questions, please contact our telephone hotline: XXXXXXXX. 

If you do not perform point-of-care testing for haemoglobin or glucose, please disregard this message. 



Yours sincerely, 
Niels Mosbaak, 
Director, CGPL 




Figure 1 Content of computer reminder. 



2. The comparison after the second intervention 
period (in which the intervention was switched 
between groups) estimates the relative effectiveness 
of a (short-term) legacy effect of ComRem over the 
direct effect of ComRem. 

3. The comparison after the third follow-up period 

(in which the ComRem intervention was discontinued 
for both groups) estimates the relative effectiveness of 
a longer-term legacy effect of ComRem over the 
short-term legacy effect of ComRem. 

Primary outcome 

Total number of split test procedures for the correspon- 
ding POCT analysis performed by the practice in a four- 
month period. In a given month, a split test procedure 
should be performed if the practice conducted a POCT 
analysis for hemoglobin or glucose. Thus, the maximum 
number of possible split test procedures for a single ana- 
lysis in a four-month period is four (i.e., for both types of 
analysis the maximum number is eight). A reminder was 
only sent if both types of analysis are missing. Therefore, 
in each of the four months periods of this study, a practice 
could have received a maximum of four reminders. 

Secondary outcomes 

1. Whether 75% of the required split tests were 
performed within the given follow-up period. This 
was defined as a high level of procedural quality 
assessment in the guidelines of the Laboratory. 

2. Whether split test procedures were performed at all 
by the practice within the given follow-up period. 

Data collection 

Data on the number of POCT tests conducted for each prac- 
tice was retrieved from the Capital Regions administrative 



database (in which information on the GPs reim- 
bursement claims for POCT tests is stored). Data on 
the number of performed split test procedures were 
retrieved from the Laboratory database. This dataset 
does not contain information immediately relatable to 
specific patients because all split tests are performed with 
an artificial identification code. Data on the number of re- 
minders were also obtained from the Laboratory. The 
Capital Region databases provided information on the 
participating practices and corresponding GPs. 

Statistical analysis 

Differences in the GP characteristics and outcomes at 
baseline between allocation groups were tested by means 
of t- tests (continuous characteristics/outcomes) and chi- 
square tests (categorical characteristics/outcomes). Ad- 
herence to external quality assessment over time relative 
to the (changing) intervention, was analyzed by Poisson 
(primary outcome) and logistic (secondary outcomes) re- 
gression with GEE methods being used to account for 
the repeated measurements. SAS, version 9.2 (SAS Insti- 
tute Inc, Cary, NC) was used for all statistical analyses. 

Results 

A total of 286 general practices conducting at least five 
POCTs for hemoglobin or glucose during the four- 
month baseline period (1 January to 31 April 2010) were 
initially included in the study. In terms of GP and prac- 
tice characteristics as well as the level of POCT- adherence, 
there were no significant differences between the two allo- 
cation groups (see Table 1). Figure 2 shows the flow of 
practices in the study. As expected dropout rates were very 
low. A total of 529 ComRems was sent to the practices 
during the two intervention periods. 
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Assessed for eligibility (n=567) 



Excluded (n=281) 

• Practices not performing the relevant 
POCT tests (n=68) 

• Practices already included in another 
POCT intervention study (n=213) 



Randomized (n=286) 



Allocated to ComRem in Period 1 (Early ComRem) 
(n=142) 



Period 1 

Practices analyzed (n=137) 
ComRems sent (n=262) 
Excluded from analysis (n=5) 

• Stopped working as GP (n=4) 

• Split testing no longer relevant (n= 1 ) 



First intervention 
period 
(Sep-Dec 2010) 



Period 2 

Analyzed (n=133) 
Excluded from analysis (n=6) 

• Stopped working as GP (n=4) 

• Split testing no longer relevant (n=2) 



Crossover 
(Jan-April 2011) 



Period 3 

Analysed (n=131) 
Excluded from analysis (n=2) 

• Stopped working as GP (n=2) 



Observation of 

legacy effect 
(May-Aug 2011) 



Figure 2 CONSORT flowchart. 



Allocated to ComRem in Period 2 (Late ComRem) 
(n=144) 



Period 1 

Practices analyzed (n=143) 
Excluded from analysis (n=l) 

• Stopped working as GP (n=l) 



Period 2 

Analyzed (n=139) 
ComRems sent (n=267) 
Excluded from analysis (n=4) 

• Stopped working as GP (n=3) 

• Split testing no longer relevant (n=l) 



Period 3 

Analysed (n=132) 
Excluded from analysis (n=7) 

• Stopped working as GP (n=3) 

• Split testing no longer relevant (n=4) 



Outcomes 

Table 2 shows the results for the main outcomes 
through the baseline period and three follow-up periods. 

In the first intervention period, the mean number of 
split tests performed in the group receiving ComRem 
group increased by a factor of three from baseline to 
end of the first follow-up (1.22 to 3.76). In the other 
group targeted by GIA only, the primary outcome meas- 
ure increased by factor of 2.12 (from 1.11 to 2.35). Com- 
pared to GIA without ComRem, GIA plus ComRem 
were significantly more effective in promoting adher- 
ence to guidelines; the number of split test performed 
was 1.40 (95% confidence interval: 1.07 - 1.83, p = 
0.0059) times higher in the Early ComRem group 
than in the Late ComRem group beyond what could 
be expected by the difference between the groups 
at baseline. 



For the secondary outcomes, the pattern was similar to 
the above in that 84% of practices in the Early ComRem 
group were performing at least one of the recommended 
split tests after the intervention against 51% at baseline 
and 60% in the late ComRem group. 

In the second intervention period, in which the interven- 
tion was switched, split test adherence in the group now re- 
ceiving ComRem increased to levels similar to those of the 
group that received ComRem in the first intervention period. 
Also, the results for the practices that received ComRem in 
the first period show a significant legacy effect of ComRem 
with adherence being close to that of the previous period. 

In the third period, for which the ComRem intervention 
was discontinued for both groups, the effect of ComRem 
decreased for both groups, but adherence (in terms of the 
primary outcome) was still significantly higher for both 
groups compared to the baseline period. 
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Table 2 Main outcomes through the baseline period and three follow-up periods 3 



Baseline period 



Early ComRem 
(n = 142) 



Late ComRem 
(n = 144) 



p-value 



ntervention 
Primary outcome 

No. of split tests (out of 8) per practice, mean (95% CI) 
Secondary outcomes 

Practices with >0% of recommended split tests performed, fraction (95% CI) 
Practices with >75% of recommended split tests performed, fraction (95% Ci) 
Period I (First intervention period) 
Intervention 
Primary outcome 

No. of split tests (out of 8) per practice, mean (SD) 
Secondary outcomes 

Practices with >0% of recommended split tests performed, fraction (95% CI) 
Practices with >75% of recommended split tests performed, fraction (95% CI) 
Period II (Cross-over, second intervention period) 
Intervention 
Primary outcome 

No. of split tests (out of 8) per practice, mean (SD) 
Secondary outcomes 

Practices with >0% of recommended split tests performed, fraction (95% CI) 
Practices with >75% of recommended split tests performed, fraction (95% CI) 
Period III (Final follow-up after ComRem has been discontinued for both groups) 
Intervention 
Primary outcome 

No. of split tests (out of 8) per practice, mean (SD) 
Secondary outcomes 

Practices with >0% of recommended split tests performed, fraction (95% CI) 
Practices with >75% of recommended split tests performed, fraction (95% CI) 



General implementation General implementation 
activities (GIA) activities (GIA) 



1.22 (0.99 - 1.49) 

0.51 (0.43 - 0.60) 
0 

(n = 137) 
GIA plus ComRem 

3.76 (3.39 - 4.16) 

0.84 (0.77 - 0.89) 
0.26 (0.19 - 0.34) 
(n = 133) 

GIA 

3.41 (2.96 - 3.94) 

0.71 (0.63 - 0.78) 
0.30 (0.23 - 0.38) 
(n = 131) 

GIA 

3.18 (2.75 - 3.66) 

0.70 (0.62 - 0.77) 
0.26 (0.19 - 0.34) 



1.11 (0.91 - 1.35) 

0.51 (0.43 - 0.59) 
0 

(n = 143) 

GIA 

2.35 (1.99 - 2.78) 

0.60 (0.52 - 0.68) 
0.13 (0.08 - 0.19) 

(n = 1 39) 
GIA plus ComRem 

3.81 (3.41 -4.26) 

0.81 (0.74- 0.87) 
0.31 (0.24- 0.40) 
(n = 132) 
GIA 

2.83 (2.43 - 3.31) 

0.66 (0.57 - 0.73) 
0.22 (0.16- 0.30) 



0.5159 



0.9039 



0.0059 



0.0000 
0.0065 



0.1030 

0.0406 
0.7614 



0.8800 



0.4707 
0.4345 



a The values in the table are the predicted values derived from the Poisson regression model (primary outcome) or the logistic regression model (secondary 
outcomes} and their 95% confidence interval calculated with GEE methods. 

b P-value pertaining to a test of difference between Early ComRem and Late ComRem (Baseline period) or a test of difference between Early ComRem and Late 
ComRem beyond the difference found in the baseline period (the three follow-up periods). 



Discussion 

Main findings and interpretations 

The results demonstrate a significant effect of computer 
reminders on all outcome measures. Furthermore, the 
developments in outcome measures during the various 
periods of the study show a strong legacy effect of com- 
puter reminders. However, while the computer re- 
minders proved to be quite effective in nudging most 
practices towards a higher degree of adherence to quality 
assessment guidelines, the reminders were ineffective 
with regard to a minority of practices. Thus, 16% and 
19%, respectively, of the practices were not performing 
any of the recommended split tests after each group had 



received the ComRem intervention. The increase in ad- 
herence to quality standards observed in the late 
ComRem group during the first period of the study may 
be ascribed to the general strengthening of implementa- 
tion activities previously mentioned. 

Strengths and limitations 

This study had several methodological strengths. First, 
the outcome measures were highly valid in being inde- 
pendent of subjective perceptions in the study popula- 
tion. Second, selection bias in the study population was 
avoided, because inclusion in the study was not based 
on voluntary participation. Third, dropout rates during 
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the intervention period were very low. Furthermore, in 
this setting ComRem seems to be a feasible intervention, 
which can easily be converted into a permanent method 
for supporting implementation of external quality assess- 
ment. Thus, the reminders employed in this study con- 
stitute a relatively simple and low-cost technology 
compared to advanced computerized decision support 
systems. However, the technical infrastructure required 
for sending automatically generated reminders straight 
to the GPs electronic patient journal obviously puts cer- 
tain limits on the immediate transferability of the inter- 
vention beyond this particular setting. Also, it could be 
argued that solely measuring process adherence repre- 
sents a limitation because this means that we do not 
know whether the actual quality of the point-of-care 
tests improved along with the increased adherence to 
procedural standards. This points to another limitation, 
namely that the quantitative data does not tell us what 
specific actions are taken by practices who wish to step 
up adherence when they receive a reminder (e.g., going 
over the correct use of the testing equipment, recali- 
brating the equipment, or changing roles and responsi- 
bilities in the practice). 

Comparison with literature 

The most recent Cochrane review on computer reminders 
[11] found that reminders generally had little or moderate 
success in changing adherence to recommended processes 
of care (although a few studies in the review showed 
stronger effects). There may be at least two reasons why 
the effects of computer reminders in this study were far 
more pronounced than reported by the majority of 
studies in the Cochrane review: 

1. While the Cochrane review focused on reminders 
appearing on the computer screen during the patient 
encounter with the objective of improving patient care 
processes and outcomes {e.g., many of the studies 
centred on prescription) [11], the objective of 
reminders in this study was to change provider 
behaviour in regards to managing laboratory tests. 
Thus, this study deals with behavioural adherence with 
regard to a relatively simple procedure (split testing), 
and it is likely that the effect of ComRem will be 
reduced if employed to support the implementation of 
more complex recommendations [14]. 

2. Using electronic reminders to support quality 
assurance is a new phenomenon in Danish general 
practice. Hence, a certain 'novelty effect' may be 
present in this study. It is possible that the effect of 
electronic reminders may decrease if they become 
more widely used, giving rise to a situation in which 
several different reminders compete for the attention 
of health professionals. 



It should be mentioned that the possible financial conse- 
quence of non-compliance stated in the reminder was prob- 
ably not taken too seriously by the doctors because they 
have previously been informed about this and because most 
of them are probably aware that restriction of payment 
has not yet been applied in response to non-compliance. 

Conclusions 

While previous studies have reported small to moderate ef- 
fects of computer reminders, the results of this study sug- 
gest that electronic reminders can be a quite effective tool 
for changing professional behavior with regard to quality 
assurance of medical equipment. Future research could 
explore: how electronic reminders are perceived by 
health professionals, e.g., whether and when they are per- 
ceived as helpful or disturbing; how the concurrent oper- 
ation of different electronic reminders may affect the 
effectiveness of each other; and the specific circum- 
stances that amplify or reduce the effectiveness of elec- 
tronic reminders. 

In terms of practical implications, the results from this 
study have encouraged the Laboratory to make com- 
puter reminders part of its routine implementation prac- 
tice — a move that attests to the feasibility of computer 
reminders in this setting. Furthermore, in continuation 
of this study, the Laboratory may attempt to identify the 
technical and organizational changes employed by some 
of the practices that improved adherence during the 
intervention. Such knowledge could be used to assist 
other practices in adopting a more systematic approach 
to quality assurance for POCT. 

Finally, and on a more general level, it should be noted 
that the increased technical possibilities of employing 
electronic reminders place a responsibility on policy 
makers and quality assurance actors to coordinate and 
prioritize the use of electronic reminders in order to 
avoid an overflow of reminders at the clinic level. 
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This study used blood samples that were not retraceable to 
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Ethical Committee for Copenhagen and Frederiksberg 
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